[ci] run e2e tests from the runner using a remote-access kubeconfig - #1948
Open
abrarshivani wants to merge 2 commits into
Open
[ci] run e2e tests from the runner using a remote-access kubeconfig#1948abrarshivani wants to merge 2 commits into
abrarshivani wants to merge 2 commits into
Conversation
Holodeck gained a kubernetes.remoteAccess option, which rewrites the generated kubeconfig's server URL to the node's public DNS name on port 6443 and chowns the file to the owner of the bind-mounted workspace. The runner user can therefore read the kubeconfig and talk to the cluster directly, which is what we needed all along. Enable that option and revert the remote-execution workaround added in NVIDIA#1763. The workflow no longer writes the SSH private key to key.pem, rsyncs the source tree into the instance, installs Go and make there, or copies logs back afterwards. It just runs make -f tests/e2e/Makefile test on the runner, as it did before the workaround. Both LOG_ARTIFACTS and LOG_ARTIFACTS_DIR are set: the Makefile uses the former for the ginkgo reports and the test suite uses the latter for its own logs, and they need to agree for the upload-artifact step to pick everything up. KUBECONFIG goes back to being a required environment variable. Falling back to ~/.kube/config only made sense while the tests ran inside the instance, and in CI a silent fallback would point at the wrong cluster rather than fail. The security group still exposes 6443 only to the auto-detected caller egress IP, and AWS_SSH_KEY is still passed to the Holodeck action, which needs it to provision the instance. Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
The step uploads ginkgo.json, but the Makefile writes its ginkgo reports
to $(LOG_ARTIFACTS)/report.json and $(LOG_ARTIFACTS)/junit.xml, so that
path has never existed. Every run logs
No files were found with the provided path: ginkgo.json.
No artifacts will be uploaded.
The reports now sit under e2e_logs/, which the archive step above already
picks up on failure, so the step is redundant as well as broken.
Signed-off-by: Abrar Shivani <ashivani@nvidia.com>
abrarshivani
marked this pull request as ready for review
August 4, 2026 20:23
Contributor
|
Thanks @abrarshivani , LGTM. Lets wait for the release and then we should be good to merge this. We can do the similar changes to gpu-operator's CI as well. |
rahulait
approved these changes
Aug 5, 2026
Contributor
Author
|
@rahulait Sounds good. Will merge this after the release. Sure, will take a look at gpu-operator's CI as well. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This reverts the remote test execution workaround from #1763 now that Holodeck can hand us a kubeconfig the runner can actually use.
Problem
#1763 was needed because Holodeck tightened the permissions on the generated
kubeconfig. The file was no longer readable from the GitHub Actions runner, so the e2e tests could not reach the cluster and we moved the whole test run into the EC2 instance instead.That workaround costs us a few things:
key.pemon the runner, so the secret exists as a file on diskmakehave to be installed and maintained inside the instance on every runSolution
Holodeck added a
kubernetes.remoteAccessoption. When it is set, Holodeck rewrites the kubeconfig server URL tohttps://<PublicDnsName>:6443and chowns the file to the owner of the bind-mounted workspace, so the runner user can read it. The security group still opens 6443 only to the auto-detected caller egress IP, and the file mode stays0600.So this PR sets
remoteAccess: trueintests/e2e/infra/aws.yamland drops the rsync/SSH scaffolding. The workflow goes back to runningon the runner, which is what it did before #1763.
The Holodeck action is already on v0.3.7 and the feature landed in v0.3.4, so no version bump is needed.
Notes
AWS_SSH_KEYis still passed to the Holodeck action, which needs it to provision the instance. Only the manual step that materialized it askey.pemis gone.KUBECONFIGgoes back to being a required environment variable in the test suite. The fallback to~/.kube/configonly made sense while the tests ran inside the instance, and in CI a silent fallback would point at the wrong cluster rather than fail.Both
LOG_ARTIFACTSandLOG_ARTIFACTS_DIRare set now. The Makefile uses the first for the ginkgo reports and the suite uses the second for its own logs, and they need to agree for the upload step to pick everything up.The second commit removes the
Archive Ginkgo logsstep. It uploadsginkgo.json, which the Makefile has never written, so every run has been loggingNo files were found with the provided path: ginkgo.json. The reports it was meant to capture now sit undere2e_logs/, which the step above it already uploads on failure.Validation
A full CI run passed on the first commit. Ginkgo ran from the runner workspace rather than inside the instance, and the suite reported
Ran 4 of 4 Specs,4 Passed | 0 Failed, matching the spec count onmain.Left as a follow-up: gpu-operator uses the same remote execution pattern and could be simplified the same way, but its logging and test environment differ enough to be worth a separate change.